Follow the User?!

Data Donation Studies for Collecting Digital Trace Data

Session 3️⃣: Data Donation Studies (Researcher Perspective)

Frieder Rodewald (University of Mannheim) & Valerie Hase (LMU Munich)

👉 Part of the SPP DFG Project Integrating Data Donations in Survey Infrastructure

Data donation study - researcher perspective

Figure. Data donation study - researcher perspective

Agenda

1. Research design & tool set-up

2. Data cleaning & augmentation, including

   📢 Task 3: Classify search terms

3. Modelling digital traces

Image by Hope House Press via Unsplash

1) Research design & tool set-up (Frieder)

Source: Image by Markus Winkler via Unsplash

Step I: Research design & tool set-up

Figure. Data donation study - researcher perspective

Step I: Research design & tool set-up

Key decisions:

• Which theoretical questions do I want to answer?
• How do I operationalize key variables via my data donation tool?
• How do I integrate the tool in surveys & recruit participants?

Step I: Research design & tool set-up

Key decisions:

• Which theoretical questions do I want to answer?
• How do I operationalize key variables via my data donation tool?
• How do I integrate the tool in surveys & recruit participants?

Step I.I Which questions do I want to answer?

This may sound silly but:

• Novel method, few empirical applications
• To date: methodological playground
• What good is a method that is not used to advance theories/empirical knowledge?

Step I: Research design & tool set-up

Key decisions:

• Which theoretical questions do I want to answer?
• How do I operationalize key variables via my data donation tool?
• How do I integrate the tool in surveys & recruit participants?

Step I.II: How do I operationalize key variables?

Choose a tool, e.g., …

• Port (Boeschoten et al., 2023) (Netherlands, different platforms)
• Data Donation Module (Pfiffner et al., 2022) (Switzerland, different platforms)
• WhatsR (Kohne & Montag, 2024) (Germany, WhatsApp)

Step I.II: How do I operationalize key variables?

• Participants “upload” data
• Local extraction, anonymization, & aggregation
• Users can delete data
• Informed consent, only then: send to researcher server

Step I.II: How do I operationalize key variables?

• Participants “upload” data
• Local extraction, anonymization, & aggregation
• Users can delete data
• Informed consent, only then: send to researcher server

Step I.II: How do I operationalize key variables?

Extraction🔎:

Figure. Filtering data - File extraction

Step I.II: How do I operationalize key variables?

Extraction🔎:

Figure. Filtering data - Python code

Step I.II: How do I operationalize key variables?

Extraction🔎:

Figure. Filtering data - Python code

Step I.II: How do I operationalize key variables?

• Participants “upload” data
• Local extraction, anonymization, & aggregation
• Users can delete data
• Informed consent, only then: send to researcher server

Step I.II: How do I operationalize key variables?

Anonymization 🙈:

Figure. Anonymization - Example of Whitelists

Step I.II: How do I operationalize key variables?

Anonymization 🙈:

Figure. Example of anonymized data

Step I.II: How do I operationalize key variables?

• Participants “upload” data
• Local extraction, anonymization, & aggregation
• Users can delete data
• Informed consent, only then: send to researcher server

Step I.II: How do I operationalize key variables?

Aggregation 🧮:

Figure. Aggregation - Python code

Step I.II: How do I operationalize key variables?

• Participants “upload” data
• Local extraction, anonymization, & aggregation
• Users can delete data
• Informed consent, only then: send to researcher server

Step I.II: How do I operationalize key variables?

Data deletion by users ❌:

Figure. Data deletion

Step I.II: How do I operationalize key variables?

This is how much “fun” testing DDTs is:

Figure. Github issues - Testing the tool

Step I.II: How do I operationalize key variables?

Key issues 🚨 (Hase et al., 2024)

• Missing documentation by platforms (e.g., file structure)
• Sudden changes in DDPs
• Differences across languages & devices
• Insufficient in-tool classification

Step I: Research design & tool set-up

Key decisions:

• Which theoretical questions do I want to answer?
• How do I operationalize key variables via my data donation tool?
• How do I integrate the tool in surveys & recruit participants?

Step I.III: How do I integrate the tool in surveys & recruit participants?

• Often: survey, then forwarding to an external site
• Less often: Integration in existing survey infrastructure (Haim et al., 2023)

Step I.III: How do I integrate the tool in surveys & recruit participants?

Step I: Research design & tool set-up

Figure. Data donation study - researcher perspective

Step II: Data cleaning & augmentation (Valerie)

Figure. Data donation study - researcher perspective

Step II.I: How do I clean and extend data?

This is how your data may look like:

Figure. Donated data - example

Step II.I: How do I clean and extend data?

This is how your data may look like:

Figure. Donated data - example

Step II.I: How do I clean and extend data?

• Manual annotation by participants during data donation
• APIs/scraping to extend collected data
• Text-as-data methods for classification

Figure. Donated data - example

Step II.II: How do I check for bias?

• Errors in representation and measurements, e.g.
  • based on systematic drop-out (Pak et al., 2022)
  • based on systematic misclassification of digital traces (TeBlunthuis et al., 2024)

👉 You know the drill: We will talk about this in session 4️⃣.

Step II: Data cleaning & augmentation

Figure. Data donation study - researcher perspective

Step III: Modelling (Valerie)

Figure. Data donation study - researcher perspective

Step III.I: How do I analyze results?

Think carefully about…

• How to create indices from different metrics (e.g., liking, sharing, or commenting on content)
• Hierarchical structure (nested in time, metrics, platforms)
• Skewed data, non-linearity

Summary: Researcher perspective 📚

• Summary: Key steps include…

  1. Research design & tool set-up
  2. Data cleaning & augmentation
  3. Modelling

• Further literature:

  • Boeschoten et al. (2022)

  • Carrière et al. (2024)